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ABSTRACT 


The objective of this work was to determine whether a direct statistical or 
stochastic relationship between the following systemic characteristics of dedicated built- 
in-test-equipment (BITE) could be derived and quantified: annual maintenance costs, 
user acceptance, operational availability, and “complexity” (defined as total number of 
sensor interfaces per system). Three systems of ascending degrees of complexity from the 
USAF F-15A/BC/D, O/A-10A, and C-5A/B/C were analyzed, and based upon raw data 
acquired from field operating units and fleet-wide maintenance data collection a model 
was constructed to derive constraints on a postulated “best-fit” interdependence between 
these four characteristics. The chief finding was that BITE reliability and minimal 
intrinsic system maintenance burden were the prime determinants of user acceptance and 
therefore system success. A corollary finding was that the number of data interfaces (or 
sensors) was mathematically irrelevant to user acceptance, suggesting that condition- 
based monitoring schemas are feasible provided that BITE system-level reliability is 
maximized with a minimal maintenance burden placed on the user community. Sensor 
redundancy to achieve this goal was the suggested method. This model may be used as an 
objective criterion for evaluating future BITE system procurement proposals, a critical 
concern for the emerging predictive/condition-based maintenance paradigms currently 
favored by the Department of Defense, NASA, and other Federal and commercial 


agencies. 
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EXECUTIVE SUMMARY 


This paper examines the utility and user acceptance of three dedicated aircraft 
built-in-test equipment (BITE) suites with respect to documented maintenance man-hour 
expenditures over a twenty-year period as well as a survey of system users. The objective 
was to determine the factor(s) that influence BITE acceptance by end-users (in this case, 
aircraft maintainers) and thereby provide design heuristics and other relevant guidance 
for future system design. BITE effectiveness is a central tenet of the emerging DoD-wide 
condition-based maintenance (CBM) paradigm, and therefore maximizing the quality and 


utility of data return from such systems is critical for the initiative's success. 


The chief finding was that user acceptance of BITE equipment is inversely related 
to the amount of effort required by users to maintain the equipment itself. BITE is 
unusual in that its primary end users are also usually the same people who maintain it, 
and perceived value of its functionality is critically dependent upon its perceived, more so 
than empirically determined, reliability. One surprising ancillary finding was that this 
user trust seemed to be statistically independent of the number of BITE-monitored 
subsystems, which is encouraging for the future of CBM but also may require greater 
investment in BITE system physical infrastructure and reliability engineering during 


system development. 


Recommended design heuristics include providing redundant sensors for critical 
weapon system functions and explicitly requiring that future BITE systems must not 
require more maintenance than the system(s) that they monitor. Future research may 
reveal other useful aspects of this issue that will improve operational availability while 
minimizing sustainment costs for all highly integrated manned and unmanned DoD 


weapons systems. 
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I. INTRODUCTION AND PROBLEM DEFINITION 


A. BITE BACKGROUND INFORMATION 


Built-in-test-equipment (BITE) as a distinct sub-system design discipline began 
its most rapid evolutionary phase primarily in the commercial aviation sector during the 
1980s, although its roots can be found in the early manual “push to test” functions used 
by many aircraft systems designed in the 1950s. Early features were designed to provide 
pre-flight or, occasionally, in-flight equipment confidence checks for aircrews in order to 
more readily and rapidly discriminate between malfunctions of indicating equipment and 
mechanical systems, or to identify true abnormal performance. Use of these features by 
maintenance personnel was an incidental byproduct that later became dominant in 
aviation aS a means to minimize procuring and maintaining ancillary ground test 
equipment. In the author's opinion, BITE is also perceived by many planners as a means 
to minimize personnel training requirements by incorporating as much self-diagnostic 
capability as possible into weapons system design. BITE is also now widely employed in 
ground, maritime, and space-borne weapons systems (though restricted to ground systems 
for the latter in its traditional sense) for both operational confidence checks and as a 


central tool for malfunction isolation and rapid restoration of full functionality. 


The Aeronautical Radio Corporation of America, usually referred to as ARINC, 
published the first standard for airborne BITE, ARINC 604 in 1985 (Spitzer, 2007). In 
addition to defining specific serial data protocols, this standard for the first time defined a 
centralized fault display system that would permit aircrews and maintainers to view 
failure messages from a wide variety of equipment at one location. Prior to this first 
attempt at standardization, the US Air Force had already developed and fielded several 


dedicated BITE systems with mixed results, which will be further explored later. 


Modern built-in-test (BIT) systems can be broadly placed under two primary 


categories based on their operational modes and methodologies: active and passive. 


Active systems initiate test routines in subsystems or components and report 


results to operators or maintainers. Active BIT is usually confined to a specific platform 


1 


system in order to prevent spurious inputs from otherwise non-function-essential 
equipment from interfering with (or disabling) the system during normal operations. A 
prime example of this design philosophy is the extensive built-in-test routine employed 
by most aircraft autopilots. However, platforms of more recent vintage such as the C-17 
do employ multi-system distributed BITE as a consequence of the highly integrated 
nature of the weapons system. The C-17 flight control system BIT requires several 
minutes for full execution and tests every critical subsystem related to aircraft control and 


engine performance on the aircraft. 


Passive systems monitor and record the performance of various platform 
subsystems, and the results are examined after mission execution for evidence of 
monitored system malfunctions as well as archived for historical trend data analysis. 
Most passive BITE does not provide a real-time display to operators during mission 
execution, and consequently its target user community is restricted to maintainers. An 


exception to this design philosophy is the C-5 MADAR system. 


The pioneering C-5A MADAR (Malfunction Analysis, Detection, and Recording) 
system described by Mash (1968) was an early attempt to implement comprehensive 
weapons system platform-level condition monitoring. As originally designed, the system 
consisted of 40 modules, or line replaceable units (LRUs) and periodically sampled 
nearly 800 test points during flight. Certain critical aircraft performance parameters such 
as engine vibration were displayed during flight on an oscilloscope and monitored by the 
flight engineer. Given its large number of components, MADAR was (and still is) 
arguably the most complex discrete subsystem on the C-5A, and perhaps unsurprisingly 
also incurred a large number of maintenance actions not only for the systems it 
monitored, but for its own upkeep. Its overall utility and added value to the C-5A mission 
was therefore open to question on many levels. Later upgrades to the system were 
apparently focused on adding additional monitoring capability (currently 2000 test 
points) and improving the both the reliability of the LRUs which provide the human- 
machine interface and presentation methods more so than improving user-level 


confidence in system accuracy. 


Figure 1. Avionics technician prepares to troubleshoot a C-5A. (From: G. Wantland, 
2007) 


This thesisexamines dedicated BITE of different types, complexity, and 
objectives across three USAF weapons systems, and derives a set of boundary conditions 
and/or figure(s) of merit that may yield the best value for warfighters during future BITE 


design efforts while simultaneously conserving system acquisition and sustainment costs. 


B. RESEARCH QUESTIONS 


At what point, and based on which factors, does maintenance of built-in-test and 
monitoring (BITE) subsystems obviate or negatively impact weapons system life-cycle 
cost and operational availability? If this question has a quantifiable answer, then the 
following design heuristic emerges: “BITE must not be more time-consuming or 
expensive to maintain than the system(s) it monitors.” An analogous statement is 
frequently used as contractual language for the acquisition of new space systems that 
replace legacy equipment, which can be paraphrased as follows: new systems must not be 
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more difficult to maintain than the old. Specifying BITE in this context is a narrowing of 
this paradigm; however, this improved focus also offers the opportunity to introduce far 


more rigorous metrics as well as design guidelines, as will be seen later. 


Is there an empirical relationship between the overall systemic complexity of 
BITE subsystems, the analogous complexity of monitored subsystems, and total life-cycle 
cost for a given integrated weapons system so equipped? This question is designed to 
determine whether a critical point, or intersection of the stated variables, exists beyond 
which BITE does not add value to weapons systems maintenance but is either neutral or a 
burden to sustainment. (‘Value’ in this context is defined as the system's capability to 
enable more rapid and accurate repair of monitored systems versus its intrinsic 
maintenance burden.) Conversely, an insufficiently sophisticated or capable BITE suite 
may add little if any value whatsoever, and again act as a negative factor for sustainment. 


Both regions of this potential operational space will be examined. 


Can a mathematical model representing interfaces and monitoring capabilities be 
constructed to optimize BITE infrastructure during the design phase to provide maximum 
operational availability while minimizing life-cycle costs? The desired end result of this 
study is a model capable of defining optimal interface levels, methods and operational 
reliability values framed by measure(s) of effectiveness for user acceptance that can 
define BITE capabilities within these constraints as a function of total weapons system 
lifecycle support expenditures. Subjective observations by the author during more than 20 
years of direct experience in aircraft maintenance led to the hypothesis that an optimal 
balance of BITE capabilities as described may be achievable, but that no such analysis 
has been performed to date that may facilitate development of systems that meet these 


criteria. 


C. RESEARCH METHODOLOGY 


After consideration of possible relevant factors, five potential decision variables 
were selected, with the presumption that interdependencies would be identified between 
at least some of these factors and provide insight as part of the research. Three Air Force 


aircraft types were selected as the platforms to be examined since extensive, long-term 
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detailed statistical maintenance and reliability data was readily available through 
restricted Internet access. Additionally, the author has had personal experience with all 
three systems and this fact facilitated acquisition and interpretation of all data. The ‘end 


users’ cited are USAF avionics maintenance personnel. 


Man-hour ratio. This was defined as the percentage of man-hours expended to 
maintain a given BITE system divided by the total number of man-hours needed to 
maintain the entire weapons system avionics suite, or in other words the fraction of direct 
labor costs directly applied to maintain a system used to improve maintenance of other 
aircraft systems. Avionics systems identified for determining total workload belonged to 
the following two-digit work unit code groups: 46, 51, 52, 55 (the group that 
encompasses dedicated BITE systems), 71, 72, and 74. The metric was obtained by 
dividing the sum of man-hours expended on maintenance of these systems over a 20-year 
time span by the man-hours expended on the code 55 group. These systems are 
maintained by enlisted personnel in a family of career fields generically referred to as 
aircraft guidance and control. Data was acquired from the USAF Reliability Engineering 
Management Information System (REMIS) with the assistance of Warner-Robins AFB 


personnel, and is provided in appendices 1, 2 and 3. 


Number of data interfaces. Defined as the total number of sensor/BITE 
connections as well as BITE LRU-to-LRU connections for a given system. Power and 
other utility interfaces were not considered since a supporting study objective was to 


qualitatively assess data return to the user. 


Perceived utility by end user. Although subjective in nature, this variable is a vital 
consideration in BITE system design. ‘Perceived utility’ in this context is defined as how 
useful the users believe the subject system to be as a troubleshooting tool. Data was 
acquired for each subject system via question #2 of a standardized survey (see Appendix 
4) that was distributed to members of the A-10, F-15, and C-5 maintenance communities. 
Please refer to the end of this section for further discussion of the survey methods 


employed. 


Perceived reliability by end user. Again subjective yet critical for application 
success, this data was also acquired via the aforementioned survey and represents the 
average of responses to question #3. ‘Perceived reliability’ is defined as the users’ core 
impression of how frequently the subject system itself malfunctions, which clearly will 


also affect the users’ level of trust in system outputs. 


Perceived maintenance burden by end user. This final subjective variable 
measures level of effort needed to resolve BITE malfunctions relative to the amount of 
time users spend resolving other aircraft system malfunctions, which is another critical 


component of operational suitability. Source data was question #4 of the survey. 


Questions 1 and 5 of the survey were ultimately not included in the data analysis. 
Question 1 measured the perceived amount of time spent by technicians maintaining the 
subject system, but given the availability of objective data from REMIS this information 
was redundant and less reliable. Likewise, question #5 was originally included to provide 
a figure of merit for the level of experience of respondents, but the data proved 
unnecessary for the analytical effort since a core assumption of BITE is that user level of 


experience is minimal. 


Variables 1 and 2 are physical system attributes that can be considered as figures 
of merit that represent gross systemic complexity. Variables 3, 4 and 5 represent the 
perceived amount of trust that the end users have in the system’s ability to perform its 
desired function as well as the level of effort needed to maintain its operation. Since 
BITE is not only used but also maintained by its end users, this variable set measures its 
user acceptance in terms of its effects on the user’s overall workload, a unique, almost 
recursive characteristic of this equipment class. Question 5 was used as a root measure of 


the respondent’s breadth of experience. 


Much of this study is centered on the core issue of user trust of a given system. 
According to Langford (2007), the concept of ‘trust’ with respect to weapons system 
performance can be defined thusly:" ...trust can be considered as follows: Trust factors 
into four uniquely definable and quantifiable variables. They are knowledge (the norms of 


operation, the expected actions, the rights and permissions granted); truth (the 


6 


verifiability of data supplied); belief (consistency of patterns and standards); and 
retribution (one’s stake or ante). Trust is the additive aggregation of knowledge, truth, 
belief, and retribution. Trustworthiness is the degree to which trust is established." The 
data acquired was selected in accordance with this definition, and the results of the study 


are framed in this context. 


All survey data was normalized by assigning numerical values to each of the five 
possible responses, summing the total for each question, and subsequently dividing this 
number by the number of samples (N), which were 9, 11 and 19 for the A-10, C-5 and F- 


15 responses respectively. 


The C-5 surveys were distributed and collected by the author in person at Travis 
AFB, CA in November 2006. A-10 surveys were distributed electronically at Pope AFB, 
NC and Davis-Monthan AFB, AZ to flightline aircraft maintainers by locally assigned 
members of the Air Force Engineering and Technical Service (AFETS), an independent 
USAF agency, and returned directly to the author. F-15 surveys were similarly distributed 
and collected by an AFETS representative at Elmendorf AFB, AK. 


All data collected was processed using Microsoft Excel® to perform all required 


calculations and generate graphical data presentations for subsequent analysis of results. 


D. STUDIED BITE SYSTEM DESCRIPTIONS 


Selected systems were chosen based on their relative complexity and scope of 
service. Based on the author's experience with approximately 20 types of USAF aircraft, 
they represent a rough continuum of interface levels and capabilities that accurately 
reflect the current state of airborne BITE functionality. Additionally, the F-15 ASP 
design philosophy is extensively emulated on other aircraft, and the A-10 TEMS system 
is also used on the KC-135R and F-117A aircraft. The least complex system in terms of 
operations, LRU count, and interfacing is the F-15 Avionics Status Panel (ASP). The A- 
10 Turbine Engine Monitoring System (TEMS) is an intermediate example for purposes 


of this analysis, and the C-5 MADAR II was the most complex system examined. 


The F-15 ASP provides “go/no-go” indications to ground maintenance personnel 
for specific aircraft systems via a simple set of black/white magnetically latched ball 
indicators mounted on the panel itself, which is located in the aircraft’s nose wheel well 
and therefore not accessible to the aircrew during flight. Dedicated system LRUs are 
limited to the status panel itself as described and a BIT reset panel in the cockpit. The 
ASP receives a serial digital data stream from the aircraft central computer (CC), and 
upon receipt of a failure status word for a given system or aircraft physical region will set 
the appropriate indicator. The ASP has 72 such indicators, but usage is controlled by CC 
operational flight program configuration; many are not used or are disabled as nuisance 


indications (Black, 2007). 


The A-10 TEMS continuously acquires and records performance monitoring data 
for the aircraft’s two TF-34-100A engines as well as selected environmental parameters 
during flight. The system is also used with an auxiliary real-time ground terminal to 
perform engine maintenance actions. No indications are provided to the pilot; indeed, no 
real-time data display exists at all unless the auxiliary terminal is employed during 
ground maintenance actions. However, stored data is downloaded from each aircraft once 
per day and input into the USAF Consolidated Engine Management System (CEMS) for 
trend analysis. CEMS also provides notification to maintenance personnel of potential 
system or engine discrepancies for specific aircraft during this process. TEMS consists of 


21 dedicated LRUs and accepts data from 10 legacy engine indication systems. 


C-5 MADAR II is a comprehensive platform status monitoring system that 
continuously acquires data on nearly every C-5 subsystem and provides a real-time 
interactive display to the aircrew during flight. MADAR consists of 40 dedicated LRUs 
that monitor approximately 2000 discrete test points throughout the aircraft. The majority 
of these LRUs are interface/signal conditioning devices that provide direct connections to 
the monitored systems and convert analog information into a serial digital data stream for 
subsequent processing by the recording/display LRU group. Like the A-10 TEMS, data is 
continuously recorded, retrieved and archived in a ground database which also provides 
malfunction notifications to maintenance personnel. One major function of the system is 
to monitor TF-39 engine vibration in real time during flight. Additionally, MADAR 

8 


provides most data inputs to the aircraft flight data recorder. Therefore, unlike other 
BITE systems described thus far, MADAR is essential for safe operation of the aircraft 


since these particular features are considered flight-critical functions. 
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Il. ANALYSIS AND INITIAL FINDINGS 


A. RESEARCH APPROACH 


The gross function of the model is to compare empirical values for the physical 
attributes of these systems ("objective variables") to measures of user acceptance 
("subjective variables"). The objective of the model is to determine relationships, if any, 
between the physical design attribute of interface quantity, documented reliability, and 


measures of user acceptance. These variables are defined as follows: 


Objective Variables (Units) 


M=man-hour ratio (BITE system maintenance hours/total avionics system 


maintenance hours) 


I=total number of data interfaces (positive integer) 


Subjective Variables (Units) 


U=perceived utility (real number between 1 and 5, large values good, source data 


was question #2 of survey) 


R=perceived reliability (real number between 1 and 5, large values good, source 


data was question #3 of survey) 


B=perceived maintenance burden (real number between 0 and 1, large values 
good, source data was the average value of question #4 of the survey divided by five to 


yield a zero to one scaling factor) 


For each system, a logarithmic plot of M versus I was derived. Likewise, R versus 
U was plotted, with the additional condition that B was used as a coefficient for U; actual 
values examined were R versus B(U). The heuristic applied to this action is that 
perceived utility of a given system for its design function is inversely proportional to the 
amount of effort required to maintain it. Further investigation also generated plots of M 


versus R and M versus B for comparison. 
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To establish a baseline for analysis, an "ideal" system was constructed for 


comparison with the following measured values: 


Table 1. | BITE system figures of merit. 


IDEAL Raw Log 
= 0.01 -2 
= 5000 3.69897 
= 5 0.69897 
= 1 0 
= 5 0.69897 
B(U) = 5 0.69897 
A-10 TEMS Raw Log 
= 0.063495358 -1.19726 
= 31 1.462398 
= 3.333333333 0.522879 
= 0.666666667 -0.17609 
= 3 0.477121 
B(U) = 2 0.30103 
C-5 MADAR Raw Log 
= 0.190651494 -0.71976 
= 2018 3.304921 
= 2.363636364 0.373581 
= 0.509090909 -0.2932 
= 2.272727273 0.356547 
B(U) = 1.157024793 0.063343 
F-15 ASP Raw Log 
= 0.042565579 -1.37094 
= 74 1.869232 
= 3.210526316 0.506576 
= 0.431578947 -0.36494 
= 3 0.477121 
B(U) = 1.294736842 0.112182 


M, the fraction of man-hours expended for system maintenance was set at 1% for 
the ideal case, almost certainly the best possible achievable value under real conditions. 
DoD weapons system operational requirement documents (ORDs) and the more recent 
capability development documents (CDD) frequently cite objective operational 
availability (A,) targets of 99% for mission-critical systems. Complex 'systems of 
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systems’ such as aircraft and space vehicles require the most accurate and dependable in 
situ BITE capability possible to achieve such goals, which in turn implies that BITE 
system maintenance expenditures should be absolutely minimal due to its presumed 
reliability. The number of interfaces (I) was arbitrarily and initially set at 5000, more than 
twice the C-5 MADAR value but reasonable for advanced next-generation equivalent 
systems. It was later found that the value of I was not significant for the model. Perceived 
reliability (R), maintenance burden (B), and system utility (U) were all set at the greatest 


possible favorable values. 


B. DATA ANALYSIS 


Graphical methods were selected as the preferred tool to examine relationships 
between these data sets. It must be emphasized that, as original research, this data has not 
been compared against any findings derived from similar studies since no equivalent 
research was found, and therefore statistical error limits are not known. Initial 
comparison of the "ideal" system to the A-10, C-5 and F-15 studied BITE systems 


yielded the following curves (10 x log y-axis used to separate values): 
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All Systems M, I, R, B(U) 


10*Log 


—e— A-10 —#—C-5 » F-15 —< ldeal 


Figure 2. BITE Performance parameters compared to ideal values. 
This comparison did not appear to be particularly useful, although it did illustrate 
two later critical points: 
M and I (the two left-most data points) should be considered, at least during 
preliminary analysis, separately from R and B(U). 


The R-B(U) line segment of the ideal system satisfied the equation y=x, while the 


real systems did not, indicating that the relationship between reliability and perceived 
user utility was not ‘simple’. 
Since the number of interfaces varied considerably among the systems, a second 


plot was generated omitting I and comparing the remaining three parameters, as shown in 


Figure 3. 
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M, R and B(U) 


—e—A-10 —-®—-C-5 —<F-15 —.— Ideal 


Figure 3. BITE performance parameters compared to ideal values. 


This plot suggested the possibility that a significant correlation may exist between 
M, the fraction of man-hours used to maintain BITE and the user acceptance measures R 


and B(U). Further examination of the ideal system's behavior was warranted. 


The relationship between M and the other variables was obviously of interest. M 
is a direct indicator of a BITE system's maintenance requirements, and understanding its 
interaction with other system attributes is important with respect to research question #1. 


Figures 3 through 5 on the following page depict these relationships: 
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Ideal Segment M vs. | 


D> —e— Series1 
_ — Linear (Series1) 
Figure 4. M-vs. I for ideal system (I=5000). 
Ideal Mvs.R 
1 
0.5 
0 
2 -0.5 —e— Series1 
J 4 — Linear (Series1) 
-1.5 
-2 
-2.5 
Figure 5. M-vs. R for ideal system. 
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Ideal M vs. B(U) 
1 
0.5 | 
0 
> 0.5 | | —e— Series1 
oo — Linear (Series1) 
-1.5 | | 
-2 aE a: 
as 


Figure 6. M-vs. B(U) for ideal system. 


This examination revealed that for M=0.01, the relationship between the decimal 


logarithm of M and those of I, R, and B(U) can be generically expressed as 


y=kx-(k+2) (eq. 1) 


Interestingly, this relationship remained valid for all values of I, suggesting that 
the number of data interfaces has no inherent effect on system maintenance time. At first 
glance that statement seems oxymoronic at best, but later discussion of design 
implications will illustrate the importance of this finding, which bodes well for future 


BITE architectures. 


As previously described, the second key relationship, R vs. B(U), is trivial in the 
ideal case (y=x), but more complex in real systems. Consider the following tabular 


comparisons of R and B(U): 


Table 2. +R and B(U) values for subject systems. 


Platform/System Log(R) Log(B(U)) 
A-10/TEMS 3.333333333 2 

C-5/MADAR 2.363636364 1.157024793 

F-15/ASP 3.210526316 1.294736842 
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Peak values of R obviously do not correlate with identical values of B(U). The 
manifest reason for this discontinuity is that B(U) represents user effort required to 
achieve the highest possible value of R and therefore an inverse relationship is expected, 
but the situation is further complicated by the fact that users assign a value judgment to 
B(U) based on the utility of data provided. To paraphrase a popular aphorism, ‘the pain 
must be worth the gain’, and accurate, reliable, useful data is the expected output of a 
BITE system. If such a system itself requires a significant amount of maintenance, then 
users will tend to regard its outputs with justifiable skepticism. The ideal case assumes 
that R and B(U) are maximized; a perfect system would require no maintenance and 
provide accurate indications at all times. Although unachievable in reality, this standard 


apparently does provide a meaningful baseline to assess real BITE system performance. 


The full data set, including graphics, is presented in Appendices 1 through 3. 
Unsurprisingly, given the fact that in an ideal system R and B(U) are equal, the system 
with the greatest level of user acceptance (the A-10 TEMS) exhibits the least significant 
linear slope when these two parameters are graphically compared. What is unexpected 
from a purely analytical viewpoint is that the least complex system in terms of dedicated 
LRUs and diagnostic resolution (the F-15 ASP) suffers from the lowest proportional user 
acceptance (greatest slope) when its perceived reliability and maintenance burden are 
similarly compared. In other words, the relationship between R and B(U) when viewed 
holistically across the sampled systems is non-linear if intrinsic physical complexity and 
M are considered independent variables. Clearly, perception is an integral component of 


user acceptance. 


C. INTERPRETATION AND SIGNIFICANCE 


The x-axis in these plots is dimensionless and merely sequential, labeling the 
parameters in the order in which they are portrayed on the graph. Despite (or perhaps 
because of) this simplicity, the relationship described by Eq. 1 remains constant, even 
when I is included despite its apparent independence from the other parameters. This 


strongly suggests that K is merely an artifact of the initial selection of a value of 0.01 for 
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M, which does not lend any intrinsic significance to the constant or the equation. 
However, this fact also suggests that the relationship as expressed is significant, and 


worthy of further investigation. 


Referring to Figure 3, it is clear that higher values of M are correlated with lower 
values of R, which in turn is associated with lower values of B(U). This ‘chain of 
casualty’ is difficult to express with true mathematical rigor because the exercise mixes 
empirical and subjective data. Nevertheless, some tentative heuristics can be derived via 


examination of the results obtained by this study. 


BITE system success depends on perceived system reliability by its end users. The 
C-5 MADAR system provides a remarkable array of in situ troubleshooting capability for 
its users, yet is not well-received by them due to the fact that a large fraction of their time 
is spent troubleshooting and repairing the system itself, which tends to introduce 
skepticism when interpreting MADAR alarms. From a design perspective, it is obvious 
that the most diverse capabilities are completely secondary to system maintenance 


reliability. 


BITE system success depends on data resolution. At the opposite end of the 
spectrum, the F-15 ASP is not well-received by its intended users because the 
information it provides is non-specific. Simple "go/no-go" indications for monitored 
multi-LRU systems can be easily dismissed as the result of transient conditions such as 
platform power fluctuations, human error, etc., and if such indications are not 
accompanied by additional evidence of malfunction such as_ operator-reported 
discrepancies then they are usually ignored with long-term damage to BITE credibility. 
Designers must consider the practical utility of provided data within the context of 
monitored system's failure mode effects/causes analyses (FMECAs) (Levitt, 2003). 


Specificity is critical for utility. 


The number of BITE data interfaces is irrelevant to user acceptance. This 
heuristic is predictive, but based on acquired data. User trust is founded on perceived 


reliability, which is a function of both BITE-provided data quality and the BITE system's 
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maintenance burden. The ideal system model provides a large number of interfaces, but 
the model's output does not change for all values of I. This characteristic may provide 


valuable guidance to designers as discussed in the following sections. 


D. RESEARCH QUESTION ANSWERS 


Although this study is of necessity quite limited in scope, preliminary answers to 


the stated research questions can be derived from this data: 


At what point, and based on which factors, does maintenance of built-in-test and 
monitoring (BITE) subsystems obviate or negatively impact weapons system life-cycle 
cost and operational availability? Results strongly indicated that M is the primary factor 
influencing weapons system life-cycle costs associated with BITE. Using a standardized 
labor cost of $50.00 per hour yields the following average annual BITE system 


maintenance expenditures for each weapon system examined: 
A-10 TEMS: $263,000 (200 aircraft, $1315 per airframe, M=6.35%) 
C-5 MADAR: $1,247,293 (94 aircraft, $13,269 per airframe, M=19.07%) 
F-15 ASP: $421,204 (750 aircraft, $561 per airframe, M=4.26%) 


It must be noted that these are direct labor costs only and do not reflect LRU 
repair costs and the corresponding support infrastructure needed to perform this work. 
Total BITE system support costs may in fact be an order of magnitude (or more) higher 
for complex systems such as MADAR. Even more importantly, no data exists to quantify 
direct costs incurred by erroneous LRU replacement (and attendant unnecessary 
organizational and Depot maintenance) from false BITE results or, conversely, from 
unnecessary or erroneous troubleshooting and component replacement performed 


because technicians did not utilize or trust BITE. 


Intangible costs that are difficult to quantify may in fact have a greater negative 
impact. High values of M were also associated with low values of R and B(U), implying 
that technicians were not convinced of the system's accuracy as a troubleshooting tool. 
Based on the author's experience, this paradigm frequently results in extensive additional 
troubleshooting of monitored aircraft system malfunctions as well as erroneous 
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component replacement. Therefore, a BITE system M value can serve as a reasonably 
reliable indicator of user acceptance and therefore system effectiveness, and M is in turn 
directly related to total system operational availability. Low values of M therefore 


increase A, and decrease total life-cycle costs. 


Is there an empirical relationship between the overall systemic complexity of 
BITE subsystems, the analogous complexity of monitored subsystems, and total life-cycle 
cost for a given integrated weapons system so equipped? Such a relationship does appear 
to exist, although ‘empirical’ may be too strong a term within the findings of this study; 
stochastic is a more accurate description. The most surprising result was that the number 
of interfaces (I) appears to have no effect on M, which is counterintuitive. More detailed 
research efforts are needed to more fully understand and quantify the large number of 
secondary variables involved. It is tempting to speculate that more effective BITE self- 
diagnostics would improve M, but this conjecture if pursued to its logical extreme might 
easily devolve into an endless amount of increasingly detailed internal self-monitoring, 
which at some point would act to increase M as even more transient conditions generate 


operator alarms. 


Can a mathematical model representing interfaces and monitoring capabilities be 
constructed to optimize BITE infrastructure during the design phase to provide maximum 
operational availability while minimizing life-cycle costs? Optimization in the sense 
originally envisioned at the beginning of this work does not seem feasible. However, the 
analysis did define M to reveal it as the most critical design parameter for system success 
due to its direct correlation with measures of user trust and independence from the 


number of data interfaces. 
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II. RESULTS AND RECOMMENDATIONS 


A. RECOMMENDATIONS FOR FUTURE DESIGNS 


The prime heuristic derived is that dedicated BITE systems must minimize user 
maintenance burden in order to gain acceptance and fulfill the system's design objectives. 
Despite the demonstrated independence of the number of interfaces to M, it is obvious 
that most malfunctions will occur at such interfaces, particularly in electrical connectors 
and other areas of discontinuity in signal paths that are exposed to external environmental 
effects. Therefore, implementing multi-path and multi-sensor redundancy in a manner 
similar to that employed for safety-critical systems such as automatic pilots (Branch, 
1998) would be a most effective strategy to assure that a given BITE architecture wins 
user acceptance. For example, the C-5 MADAR engine vibration monitoring suite could 
be augmented by adding another full set of sensors to each engine, thus providing an 
inherent "reality check" to sensor readings and preventing or minimizing 
unnecessary/erroneous maintenance, which in this instance can include full replacement 


of suspect engines. 


An important secondary finding was that data resolution and return was positively 
correlated with user acceptance, though modulated by perceived system reliability. As 
previously discussed, the F-15 ASP 'go/no-go' indication method apparently did not yield 
sufficiently decisive data to win confidence in the veracity of the results. Conversely, the 
C-5 MADAR provides not only threshold exceedance alarms but also real-time data 
usually presented in the form of 0 to 5 volt DC conditioned and scaled signal levels. 
Despite this unusually versatile troubleshooting capability, the system suffers from poor 
user acceptance; in the author's opinion, the sheer number of MADAR alarms commonly 
generated leads many technicians to disregard them as transient phenomena in much the 
same way that the F-15 ASP panel is seen as irrelevant in some respects, and leads to a 


dismissive attitude towards the system's capabilities. 


The most successful system examined, the A-10 TEMS, provides not only 


recorded in-flight data for later analysis but also (via use of secondary ground support 
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equipment commonly known as the digital display unit, or DDU) the ability to examine 
real-time data presented in engineering units. This design philosophy seems to increase 
user acceptance in two fundamental ways. Since the DDU is portable, it accuracy can be 
readily checked by connecting the device to another aircraft, or attaching a different unit 
to a suspect aircraft. Secondly, the DDU's displays closely correspond with those of the 
systems monitored by TEMS, the aircraft engines; interpolation of TEMS data is greatly 


minimized thereby. 


A corollary to the prime heuristic that dedicated BITE systems must minimize 
user maintenance burden in order to gain acceptance and fulfill the system's design 
objectives is that increased development and procurement costs needed to provide the 
requisite sensor redundancy will yield reduced sustainment costs over the lifetime of the 
weapons system. Due to highly variable budget environments, most Department of 
Defense systems across all mission areas usually remain in operational use well beyond 
their original design lifetimes. Additionally, equipment such as space mission control 
systems are often designed for "backwards compatibility" with legacy satellite 
constellations. A recent example is the Advanced Extremely High Frequency (AEHF) 
Satellite Mission Control System (ASMCS), which will replace the Milstar Satellite 
Mission Control System (SMCS) (Branham, 2006). Accurate BITE capability that is 
trusted implicitly by its users is urgently needed both due to the nature of the equipment's 
function and the need to restore it quickly as well as the fact that such a system must 
interface with two or more generations of technology in terms of spacecraft controlled. In 
the AEHF example, the need to distinguish latent interface incompatibilities, software 
problems, and operator errors from genuine equipment malfunctions is overwhelmingly 
critical and time-sensitive during both peacetime and wartime military operations since 
potential loss of a strategically essential communications satellite would have a 


devastating impact on all DoD functions. 


B. IMPLICATIONS FOR ADVANCED BITE ARCHITECTURES 


Condition-based maintenance (CBM) is an emerging concept for most DoD 


weapons systems. The central concept is to continuously monitor subsystem performance 
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and execute preventative maintenance actions prior to the occurrence of non-mission- 
capable failures (US Department of Defense, 2006). Aircraft engine trend monitoring in 
the commercial sector is an early implementation of CBM, and USAF is deploying a 
more comprehensive architecture known as CFRS (computerized fault reporting system) 
which will monitor the F-15 and C-130 fleets. Clearly, the goals of CBM cannot be 
realized if platform BITE lacks reliability and incurs a large maintenance burden, or, in 
other words, exhibits a high value of M. Not only would consequent low levels of user 
acceptance eventually drive such systems into limited use, but overall sustainment costs 
would increase disproportionately, especially if ancillary technical data is not procured 
due to mandates to use perceived--at least at higher echelons--infallible BITE systems, 
and consequently lower levels of training are provided to end users in a misguided belief 


that a given weapon system "can troubleshoot itself". 


Arguably, this situation is the case with the C-5 MADAR. Despite 
troubleshooting trees that can stretch into literally hundreds of steps, the system if used 
mechanistically as intended in accordance with published technical data absent of 
inherent experience and judgment rarely produces decisive conclusions. Indeed, the sheer 
magnitude of effort required to perform these procedures verbatim frequently discourages 
technicians from performing them in their entirety, and therefore again defeats the 


functional utility of the system. 


Sensor redundancy in proportion to monitored function criticality combined with 
parsimonious selection of other subsidiary monitored subsystems is therefore the only 
reasonable design strategy apparent to avoid these highly undesirable circumstances and 


consequences. 


These design observations may yield significant improvement in future BITE 
architectures. However, one of the most intractable problems revealed by the study was 
the fact that users and maintainers of BITE are often the very same individuals, which 
conceivably can incur a lack of oversight or objective evaluation in terms of system 
utilization and performance. It would be absurd to establish a completely distinct Air 
Force Specialty Code (AFSC) or Military Occupational Specialty (MOS) for other 


services dedicated to BITE system maintenance, but some level of separation must be 
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achieved between the user and operator roles in order to decouple them and prevent 
undesirable negative feedback loops to become established. One means of doing so 
would be to dedicate selected maintenance personnel to BITE maintenance only for 
limited periods. This method would both concentrate system knowledge for the detailed 
personnel and provide objective, rather than subjective, analysis of malfunctions, 
hopefully resulting in improved user confidence and subsequently better BITE system 


utilization. 


The only practical alternative to BITE is specialized test equipment, from the 
perspective of system maintainers. Such devices themselves require extensive logistics 
support throughout their life-cycle, and therefore military acquisition efforts generally 
tend to minimize development of this equipment. Furthermore, test equipment can be 
cumbersome to use, and this characteristic therefore may tend to discourage its 


employment. 


From the perspective of system operators such as aircrew members and satellite 
controllers, the unique ability of BITE to provide immediate verification of system status 
is invaluable, and cannot be replaced by any other schema. It is essential for users to have 
some indication of system status before committing to operational decisions, and external 


test equipment cannot satisfy this requirement. 


C. CONCLUSION 


Although passive stand-alone BITE may eventually prove to be a transient phase 
in weapon system design, BITE capability as a paradigm is here to stay, and the lessons 
learned in this study are equally applicable to such alternative architectures. There is a 
relatively fine line between truly adding value and merely adding unnecessary labor and 
sustainment expense through employment of such systems, defined almost exclusively by 


the derived indicator M. 


Methods for obtaining lower values of M must include both physical design 
considerations and development/adoption of user policies intended to minimize the field- 
level maintenance burden. As previously discussed, redundant sensors for critical 


platform data acquisition would allow cross-checking of indicated malfunctions and 
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thereby improve user confidence in BITE results. However, adding interfaces and 
equipment must also increase the potential for local system malfunctions and increase the 


system's maintenance burden. 


From a policy perspective, if critical sensors are triple or quadruple-redundant 
then repairs of single or double sensor failures could be deferred until the next periodic 
Depot maintenance period. Removing this additional M contribution from the field user's 
area of responsibility would facilitate a refocus on the subject BITE system's core 


functionality and therefore improve overall platform maintenance. 


For applications such as unmanned spacecraft ground control, redundant fault 
detection architectures (software or hardware based) are even more critical. Restoring 
functionality as rapidly as possible is of paramount importance and is operationally 
equivalent to an aircrew reacting to an in-flight emergency. Despite the fact that there is 
no potential for direct loss of life in these situations, loss of a space vehicle usually 
cannot easily or rapidly be redressed, and such losses invariably incur a large negative 


impact on the system's user community. 


Additionally, simplifying troubleshooting procedures to the greatest degree 
possible would again make a significant contribution to BITE employment. Onerous, 
lengthy procedures are obviously non-quantifiable but important negative influences on 
system acceptance that may act to increase M based on the author's experience, since 


‘collateral’ findings of BITE anomalies frequently occur during such efforts. 


These suggestions are only some of the possible applications of the findings of 
this research. Further investigation is likely to reveal additional methods to refine BITE 
system performance from a holistic perspective. The main conclusion of this study is that 
obtaining values of M as close as possible (or ideally lower) than 0.01 will provide truly 
useful BITE performance and greatly reduce sustainment costs over the operational 


lifetime of DoD weapon systems. 
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APPENDIX A. A-10 TEMS DATA 


Table 3. © A-10 REMIS data(April 1987-April 2007). 


System Maintenance Hours 
01 0.0 
02 0.0 
03 0.0 
04 0.0 
05 0.0 
06 0.0 
07 0.0 
09 0.0 
11 721433.6 
12 250983.1 
13 689904.7 
14 757789.8 
23 647693.9 
24 84716.0 
41 176759.6 
42 259270.7 
44 121131.8 
45 125130.5 
46 576997.6 
47 67975.1 
49 30740.3 
51 301701.3 
52 58629.1 
5D 105201.3 
62 98719.7 
63 98126.5 
64 28226.4 
65 69180.3 
69 24081.2 
71 108180.9 
72 10109.8 
74 496014.5 
75 764069.6 
76 346090.7 
91 16935.6 
96 938.7 
97 62197.5 
99 121.9 
AA 47.6 
AP 13.0 
XX 32.0 
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Appendix A. A-10 data (continued). 
Table 4. TEMS survey results. 


N=9 

A B C D 
Question 1 4 2 1 1 1 
Question 2 4 1 4 
Question 3 1 2 5 1 
Question 4 1 3 4 1 
Question 5 6 2 1 


Normalization (A=5, B=4, C=3, D=2, 


E=1) Averages Comments 
A B C D E 
>20% time resolving TEMS 
20 8 33 2 1 3.78 discrepancies 
Sometimes use TEMS to troubleshoot 
0 16 3 8 0 3.00 other systems 
5 8 15 2 0 3.33 Medium trust in system accuracy 
5 12) 12 0 1 3.33 Medium repair difficulty 


Most respondents have no experience 
30 8 3 0 0 4.56 on other acft types 
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Appendix A. A-10 data (continued). 


Figure 7. 


Figure 8. 


A-10 M vs. | 


—e— Series1 
— Linear (Series1) 


A-10 Mvs. I. 


A-10 Mvs.R 


—e— Series1 
— Linear (Series1) 


A-10 Mvs. R. 
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Figure 9. A-10 M vs. B(U). 


A-10 M vs. B(U) 


—e— Series1 


— Linear (Series1) 


A-10 R vs. B(U) 


—e— Series1 
— Linear (Series1) 


Figure 10. A-10 Rvs. B(U). 
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APPENDIX B. C-5 MADAR DATA 


Table 5. C-5 REMIS data (April 1987-April 2007). 
C-5A C-5B C-5C 
System Maint. Hrs. System Maint. Hrs. System Maint. Hrs. 

01 0.0 | 01 0.0 | 01 0.0 
02 0.0 | 02 0.0 | 02 0.0 
03 0.0 | 03 0.0 | 03 0.0 
04 0.0 | 04 0.0 | 04 0.0 
05 0.0 | 05 0.0 | 06 0.0 
06 0.0 | 06 0.0 | 07 0.0 
07 0.0 | 07 0.0 | 09 0.0 
09 0.0 | 09 0.0 | 11 34914.7 
11 1581203.5 | 11 721828.6 | 12 6452.4 
12 518756.8 | 12 226380.1 | 13 30347.3 
13 1155500.8 | 13 697184.1 | 14 17146.7 
14 681976.2 | 14 425145.3 | 23 29783.0 
23 1249294.8 | 23 725908.0 | 24 3213.1 
24 129121.0 | 24 85663.0 | 41 7602.4 
41 424502.6 | 41 213185.1 | 42 5305.5 
42 263791.5 | 42 172091.0 | 44 6240.1 
44 333337.4 | 44 185115.7 | 45 5585.9 
45 346273.0 | 45 177092.4 | 46 9005.6 
46 545152.0 | 46 273657.3 | 47 820.0 
47 84449.3 | 47 36360.2 | 49 5852.7 
49 256473.0 | 49 151935.0 | 51 5454.8 
51 265483.3 | 51 164570.0 | 52 4648.0 
52 276178.7 | 52 161455.0 | 55 5885.0 
55 332546.6 | 55 160485.8 | 57 21.3 
57 121.8 | 57 1794.5 | 59 1001.7 
59 28623.0 | 59 25107.7 | 61 716.8 
61 52808.6 | 61 27420.7 | 62 193.6 
62 26729.5 | 62 18124.0 | 63 440.4 
63 27685.5 | 63 14981.8 | 64 1773.5 
64 122614.7 | 64 72570.2 | 65 919.5 
65 37205.0 | 65 20948.4 | 66 927.4 
66 54177.5 | 66 20528.9 | 68 6.0 
68 509.8 | 68 512.7 | 69 17.8 
69 3226.4 | 69 1850.6 | 71 1010.2 
71 44173.3 | 71 22238.4 | 72 3793.9 
72 209716.5 | 72 142205.4 | 76 22.3 
76 607.2 | 76 27082.0 | 82 149.1 
82 26.2 | 82 3316.8 | 91 652.8 
91 103918.9 | 91 51981.5 | 97 416.3 
97 24730.7 | 97 9356.4 

AA 1.0 | ¥5 433.0 

Y5 18.8 | ZZ 2.0 
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Appendix B. C-5 data (continued). 


Table 6. © MADAR survey results. 


N=11 

A B D E 
Question 1 2 6 1 2 
Question 2 1 3 5 2 
Question 3 5 5 1 
Question 4 1 3 7 
Question 5 2 2 1 4 2 


Normalization (A=5, B=4, 


C=3, D=2, E=1) Averages Comments 
A B C D E 
>20% time resolving MADAR 
10 24 3 4 0 3.73 discrepancies 
Often use MADAR to troubleshoot other 
0 4 9 10 2 2.27 systems 
0 0 15 10 1 2.36 Medium/low trust in system accuracy 
5 0 9 14 0 2.55 Medium/low repair difficulty 


Most respondents have experience on at 
10 8 3 8 2 2.82 least 2 other acft types 
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Appendix B. C-5 data (continued). 


Log 


C-5 M vs. | 


—e~— Series1 
—— Linear (Series1) 


Figure 11. C-5Mvs.I 


Log 


C-5 Mvs.R 


—e— Series1 
—— Linear (Series1) 


Figure 12. C-5 Mvs. R. 


35 


C-5 M vs. B(U) 


—e— Series1 


— Linear (Series1) 


Figure 13. C-5 Mvs. B(U). 


C-5 R vs. B(U) 


—e— Series1 
— Linear (Series1) 


Figure 14. C-5 Rvs. B(U). 
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APPENDIX C. F-15 ASP DATA 


F-15 REMIS data (April 1987-April 2007). 


Table 7. 
F-15A 
01 0.0 
02 0.0 
03 0.0 
04 0.0 
05 0.0 
06 0.0 
07 0.0 
09 0.0 
11 484804.8 
12 127919.6 
13 314846.8 
14 233442.8 
1A 0.0 
23 697271.8 
24 258583.1 
28 7.3 
41 160550.3 
42 110542.7 
44 69525.8 
45 152660.3 
46 366771.8 
47 25327.1 
49 43579.9 
51 146351.6 
52 86688.8 
55 30628.9 
57 52963.1 
61 1891.9 
63 113995.4 
65 86388.3 
71 130026.8 
72 8.5 
74 679123.4 
75 240319.7 
76 351390.4 
91 2774.7 
92 605.4 
97 39918.6 
99 1828.5 


ois 


F-15B 
01 0.0 
02 0.0 
03 0.0 
04 0.0 
05 0.0 
06 0.0 
07 0.0 
09 0.0 
11 123312.9 
12 50569.3 
13 89792.7 
14 60132.3 
23 127040.5 
24 50023.6 
41 37188.0 
42 26040.3 
44 17876.6 
45 36551.8 
46 73099.6 
47 8517.8 
49 6575.8 
51 37275.5 
52 18160.8 
53 0.2 
55 5592.3 
57 13258.9 
61 400.9 
63 30845.3 
65 16743.4 
71 27455.5 
72 92.0 
74 135221.0 
75 54599.1 
76 28122.3 
91 1865.8 
92 239.1 
97 16680.3 
99 3388.0 


Appendix C. F-15 data (continued). 


F-15C 
01 0.0 
02 0.0 
03 0.0 
04 0.0 
05 0.0 
06 0.0 
07 0.0 
08 0.0 
09 0.0 
11 1702360.9 
12 431346.1 
13 1173499.4 
14 825112.9 
23 1313987.9 
24 831837.4 
28 186.1 
41 524290.7 
42 495251.2 
44 321795.9 
45 466743.7 
46 1156392.7 
47 140682.4 
49 70847.8 
51 408060.6 
52 281959.1 
53 87.8 
55 111710.6 
57 128623.4 
61 181.8 
63 413791.1 
65 272157.7 
71 449134.2 
72 134.2 
74 2573993.1 
75 1060629.4 
76 1954716.7 
77 24.2 
82 185.8 
91 16204.9 
92 504.6 
97 77904.9 
99 625.5 
AA 22.5 
ZZ 107.2 
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F-15D 
01 0.0 
02 0.0 
03 0.0 
04 0.0 
05 0.0 
06 0.0 
07 0.0 
09 0.0 
11 352467.8 
12 134693.5 
13 228223.5 
14 157345.1 
21 0.0 
23 240091.7 
24 134408.3 
28 7.0 
41 81740.1 
4? 84566.7 
44 57066.6 
45 77931.9 
46 183126.5 
47 28119.4 
49 9854.1 
51 80349.1 
52 48232.3 
53 436.9 
55 20549.9 
57 23518.1 
61 104.8 
63 95276.3 
65 41720.4 
71 77712.7 
72 15.0 
74 413003.9 
75 168492.6 
76 157290.0 
77 38.0 
82 12.0 
91 4993.2 
92 445.3 
97 23750.1 
99 911.1 
AA 0.0 
ZZ, 61.2 


Appendix C. F-15 data (continued). 


Table 8. ASP survey results. 


N=19 

A B C D 
Question 1 11 4 2 1 1 
Question 2 2 2 10 4 1 
Question 3 1 6 8 4 
Question 4 1 8 3 7 
Question 5 7 9 1 2 


Normalization (A=5, B=4, 
C=3, D=2, E=1) Averages Result 


55 16 6 2 1 4.21 <20% time resolving ASP discrepancies 


Sometimes use ASP to troubleshoot other 


10 8 30 8 1 3.00 systems 
5 24 24 8 0 3.21 Low/medium trust in system accuracy 
0 4 24 6 7 2.16 Low repair difficulty 


Most respondents have experience on at 
35 36 3 0 2 4.00 least one other acft type 
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Appendix C. F-15 data (continued). 


F-15 M vs. 1 


F-15 Mvs.R 


—e— Series1 


—— Linear (Series1) 


—e— Series1 
—— Linear (Series1) 


Figure 16. F-15 Mvs.R. 
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F-15 M vs. B(U) 


Figure 17. 


F-15 R vs. B(U) 


—e— Series1 


— Linear (Series1) 


—e— Series1 
— Linear (Series1) 


Figure 18. F-15 Rvs. B(U). 
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APPENDIX D. SYSTEM SURVEY 


[SUBJECT AIRCRAFT/SYSTEM] QUESTIONNAIRE 
Ver 2- 24 Apr 07 


1. If you are an avionics technician, how much of your time doing avionics 
work is/was spent on [subject system] discrepancies? 
a. 0%-20% c. 40%-60% e. 80%-100% 
b. 20%-40% d. 60%-80% 


2. How often do you use [subject system] to troubleshoot other aircraft 


systems? 
a. Never c. Sometimes e. Very often 
b. Rarely d. Often 


3. How much do you trust [subject system] readings when troubleshooting other 


systems? 
a. Not at all c. Somewhat e. Completely 
b. A little d. Quite a bit 


4. Compared to other [subject aircraft] avionics systems, how difficult is [subject 
system] to troubleshoot & repair? 


a. Much more c. Medium/no difference e. Easy 
b. A little more d. Not as bad as some others 

5. How many other types of aircraft have you worked? 
a. Zero c. Two e. Four or more 


b. One d. Three 


Please write your rank and AFSC, including skill level (3, 5, 7)- names aren’t needed-and 
return this to Nick Previsich. Thanks! 
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